WAIC and cross-validation in Stan∗
نویسندگان
چکیده
The Watanabe-Akaike information criterion (WAIC) and cross-validation are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model. WAIC is based on the series expansion of leave-one-out cross-validation (LOO), and asymptotically they are equal. With finite data, WAIC and cross-validation address different predictive questions and thus it is useful to be able to compute both. WAIC and an importance-sampling approximated LOO can be estimated directly using the log-likelihood evaluated at the posterior simulations of the parameter values. We show how to compute WAIC, IS-LOO, K-fold cross-validation, and related diagnostic quantities in the Bayesian inference package Stan as called from R.
منابع مشابه
Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
Leave-one-out cross-validation (LOO) and the widely applicable information criterion (WAIC) are methods for estimating pointwise out-of-sample prediction accuracy from a fitted Bayesian model using the log-likelihood evaluated at the posterior simulations of the parameter values. LOO and WAIC have various advantages over simpler estimates of predictive error such as AIC and DIC but are less use...
متن کاملBayesian Cross Validation and WAIC for Predictive Prior Design in Regular Asymptotic Theory
Prior design is one of the most important problems in both statistics and machine learning. The cross validation (CV) and the widely applicable information criterion (WAIC) are predictive measures of the Bayesian estimation, however, it has been difficult to apply them to find the optimal prior because their mathematical properties in prior evaluation have been unknown and the region of the hyp...
متن کاملApproximating Cross-validatory Predictive Evaluation in Bayesian Latent Variables Models with Integrated IS and WAIC
Abstract: A natural method for approximating out-of-sample predictive evaluation is leaveone-out cross-validation (LOOCV) — we alternately hold out each case from a full data set and then train a Bayesian model using Markov chain Monte Carlo (MCMC) without the held-out; at last we evaluate the posterior predictive distribution of all cases with their actual observations. However, actual LOOCV i...
متن کاملSlac Pub-3013 Stan-orion 003 July 1982 (ml Smoothing of Scatterplots*
A variable span scatterplot smoother based on local linear fits is described. Local cross-validation is used to estimate the optimal span as a function of abscissa value. A rejection rule is suggested to make the smoother resistant against outliers. Computationally efficient algorithms making use of updating formulas and corresponding FORTRAN subroutines are presented.
متن کاملWaic and Wbic Are Information Criteria for Singular Statistical Model Evaluation
Many statistical models and learning machines which have hierarchical structures, hidden variables, and grammatical rules are not regular but singular statistical models. In singular models, the log likelihood function can not be approximated by any quadratic form of a parameter, resulting that conventional information criteria such as AIC, BIC, TIC or DIC can not be used for model evaluation. ...
متن کامل